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ABSTRACT 

Paulo Freire, bell hooks, and Ira Shor have been criticized 
as theorists not concerned with the institutional realities of American 
education (e.g., grades and standardized exams). This paper argues that these 
teacher-researchers speak to the issues encountered in developing electronic 
portfolios in high schools and colleges. The imposition of high-stakes 
assessments from outside of the classroom (often legislated by state 
governments and enforced by state departments of education) creates 
environments where concerned teachers must teach toward the tests. Yet in 
classrooms where computer-mediated writing occupies a good deal of the 
students’ time, more and more teachers and researchers are finding that 
student creativity and risk-taking do not directly correspond to standardized 
assessment. The most recent advances in electronic portfolio assessment, 
however, demonstrate that communication-based writing evaluation can be 
developed not only in individual classrooms, but also across university 
writing programs or throughout state education systems. This paper briefly 
sketches the practices developing in electronic portfolio systems and shows 
how these methods of assessment address concerns about validity and how they 
change the tenor of the conversation when discussing reliability. Drawing on 
Brian Huot ' s- ideas in "Toward a New Theory of Writing Assessment, " the paper 
suggests that • electronic portfolio assessment can both reinvigorate critical 
pedagogy and can benefit by addressing the challenges raised by Freire, hooks 
and Shor. Contains 20 references. (Author/RS) 
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Abstract: Paulo Freire, bell hooks, and Ira Shor have been criticized as theorists not concerned with the 
institutional realities of American education (e.g., grades and standardized exams). I argue that these 
teacher-researchers speak to the issues encountered in developing electronic portfolios in high schools and 
colleges. 

The imposition of high-stakes assessments from outside of the classroom — often legislated by 
state governments and enforced by state departments of education — creates environments where concerned 
teachers must teach toward the tests. Yet in classrooms where computer- mediated writing occupies a good 
deal of the students’ time, more and more teachers and researchers are finding that student creativity and 
risk-taking do not directly correspond to standardized assessment. The most recent advances in electronic 
portfolio assessment, (e.g., Hart-Davidson, RPI; Rice, Texas Tech; Sy verson, Texas; and Yancey, - 
Clemson) however, demonstrate that communication-based writing evaluation can be developed not only in 
individual classrooms but also across university writing programs or throughout state education systems. 

In this presentation, I will briefly sketch the practices developing in electronic portfolio systems. I 
will show how these methods of assessment address concerns about validity and how they change the tenor 
of the conversation when discussing reliability. Drawing on Brian Huot’s ideas in “Toward a New Theory 
of Writing Assessment,” I suggest that electronic portfolio assessment can both reinvigorate critical 
pedagogy and can benefit by addressing the challenges raised by Freire, hooks and Shor. 
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The Critique of Critical Pedagogy: Richard E. Miller “Arts of Complicity” 

One of the most articulate challenges to the practice of critical pedagogy is 
Richard E. Miller’s “Aits of Complicity.” Miller argues that the veneration of Paulo 
Freire and critical pedagogy within postsecondary composition studies has as much to do 
with teachers’ desires to see themselves as resisting. corporations and dominant forms of 
cultural production as it does with students’ genuine, contextualized needs. Miller writes, 
Far from being powerless, as teachers who have years of experience in this 
frequently capricious and indifferent system for distributing social privilege, we 
are actually very well positioned to assist out students in acquiring the skills 
necessary for persisting in the ongoing project of navigating life in a bureaucracy. 



BEST COPY AVAILABLE 



2 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

□ This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction qualiiy. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



For Miller, critical pedagogy in writing studies sells out students in favor of ideology. 

Or, to put it less bluntly, Miller hopes that teachers will consider the ways in which they 
can engage students in developing skills for “the ongoing project of navigating life in a 
bureaucracy.” 

If we look at the methods of writing assessment that are emerging in electronic 
portfolio systems, we see a glimmer of hope — we see the possibility that the interactive, 
student-centered learning advocated for by proponents of critical pedagogy can coincide 
with the development of communication skills that reflect the complexities of student- 
generated, computer-mediated compositions. 

Distributive, Interactive, Descriptive and Situated Evaluation: Emerging Methods of 
Assessment in Electronic Portfolio Systems 

The developing body of work on electronic writing assessment suggests that there 
is the possibility — perhaps even the necessity — of creating methods of distributive, 
interactive, descriptive and situated evaluation that reflect the risks and complexities of 
student-generated, computer-mediated compositions. Each of these methods of 
assessment reflects a growing understanding of evaluation as a process of 
communication. Distribution acknowledges that different readers will respond 
differently to texts and does not smooth out these differences but instead incorporates 
multiple and distinct responses. Interaction argues that assessment should include 
feedback and negotiation. Description allows a reader to respond from her experience as 
a reader instead of through a filter of criterion- or standards-based assessment. And 
situated readings actively work to include — rather than exclude — social contexts. None 
of these methods outline discrete skills or areas for measuring writing ability; rather as an 
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amalgam of approaches they sketch the groundwork for treating evaluation as 
communication. In this manner, evaluation and assessment no longer impose abstract 
external standards upon students, but rather, in a classical move of critical pedagogy, 
challenge students to reflect on their learning and to meet the standards generated within 
the contexts of communication. 

Assessment Theory in Composition Studies: Brian Huot’s “Toward a New Theory 
of Writing Assessment” 

Between 1996 and 2001 , discussions about electronic portfolios began to connect 
assessment and computer-mediated communication in K-12 and higher education circles. 
Within critical pedagogy, assessment and evaluation are often seen as secondary issues to 
discussions about promoting social change through educational practices. However, the 
questions of literacy practices and social values that writers such as Paulo Freire, bell 
hooks, and Ira Shor explore bear directly on questions of evaluation and assessment. For 
instance, we see the convergence of these questions in the work of Brian Huot on 
context-based assessment. Huot draws on the work of measurement experts such as 
Pamela A. Moss, Samuel Messick, and Lee Cronbach. In a study of how the material 
conditions of word processing affect writing assessment systems, I noted that Huot shows 
how validity has begun to move away from an empiricist notion of objectively 
determined accuracy toward acknowledging “the social construction of 
knowledge” (“Towards” 550). Huot quotes Cronbach’s claim that validity “‘must 
link concepts, evidence, social and personal consequences and values’ (4)” to 
demonstrate the redefinition of validity within the assessment community (550). 
This work within the assessment community suggests links with literacy scholars’ 



ERIC 



4 




research and the localized writing testing programs used at the University of 
Pittsburgh and Washington State University (553-554). For Huot, assessment 
theory, literacy and reading theory and some composition testing programs 
suggest “emergent methods” of assessing writing that stretch older, accepted 
definitions of valid, reliable, and authentic assessment. 

These emergent methods can be view under a new theoretical umbrella, 
one supported by evolving conceptions of validity that include the 
consequence of the tests and a linking of instruction and practical purposes 
with the concept of measuring students’ ability to engage in a specific 
literacy event or events.... These methods are sensitive to the importance 
of interpretation inherent in reader response and psycholinguistic theories 
of reading. (Huot 561) 

Despite the six years that have passed since Huot articulated the basis for a “new 
theory of writing assessment,” the uses of validity and inter-rater reliability in the 
literature on writing assessment have remained bound by their traditional 
connotations. Validity tends to mean that the assessment tool matches the content 
to be covered; and, reliability tends to mean that different readers will produce the 
same score for the same essay or type of essay across contexts. (Whithaus) 

In print-based writing portfolios, authentic assessment is always situated — at least to 
some extent — sometimes descriptive, occasionally interactive, but almost never 
distributive. Assessments of student work in computer-mediated environments cannot 
rely on non-distributive, teacher-centered methods of evaluation if they hope to represent 
the complexities of student work. Assessment in computer-mediated composition spaces 
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requires that we continue to push the emerging definitions of validity and reliability. It 
also requires that we rethink the methods of authentic assessment associated with print- 
based portfolios. The value of a text is tied to its use, its context, and its past; the value 
of an assessment system is also tied to its use, its context, and its past. The evaluation of 
computer-mediated compositions offers us a chance to rethink our methods of assessment 
and to bring these assessments into a contextualized process of communication. 



Assessment Theory in K-12 Education: Eva Baker’s Understanding Educational 
Quality 

An interest in context-based assessment is not limited to higher education, but 
rather speaks directly to concerns within K-12 assessment and K-12 teaching 
communities. The increasing numbers of standards-based assessments imposed on 
students and teachers during the last five years are shaping our students’ experiences of 
education. These experiences socialize students into passive roles and lower their 
expectations for the relevance of literacy education to their daily lives. 1 A move toward 
communication-based assessments in K-12 assessment could enable students to see 
themselves as active — rather than passive — agents in the construction of literacy, 
learning practices, and social values. 

In her discussion of K-12 testing, Eva Baker notes that current systems and 
theories of assessment do “not meet our expectations for guiding practice and improving 



1 Ira Shor has noted that 

language arts are constant requirements for students from elementary grades through college, 
making language instruction the biggest and most closely-watched enterprise in mass education. 
From childhood through early adulthood, official language arts help to socially construct how 
students see the world and act in it (Pattison; Rouse). This socialization through curriculum (what 
Paulo Freire called “the banking models” of pedagogy) uses assessment and instruction as vast 
“sorting machines,” to borrow Joel Spring’s metaphor.... The tradition of complaint in first-year 
college writing is a product of the contention faced by the status quo in reproducing itself in each 
new generation. (“Illegal Literacy,” 105) 
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learning” (15). As an expert on assessment, she makes it clear that her complaints about 
testing do not emerge from the usual ideological positions which resist “the capital letters 
TESTING INDUSTRY” and valorize “the wisdom and accuracy of classroom teachers’ 
judgments” (15). 2 Even from her position within the testing community, Baker urges that 
current methods of assessment are failing to meet the needs of students, teachers, 
communities, and policymakers because they center on measurements of discrete skills. 
System validity and multipurpose testing are “heretical” ideas for the assessment 
community (16). Research in computer-mediated composition and electronic portfolios 
suggests that students draw on a variety of communication skills and interact with other 
writers and audience members in multifaceted ways. For the evaluations — either 
classroom-based or large-scale — of computer-mediated compositions to be valid and 
authentic, then, our assessment tools must take effective communication as their 
benchmark instead of discrete skill-based standards. In turn, this shift in assessment 
methods and criteria returns us to the questions of literacy activities and social values that 
underlie the work of Freire, hooks, and Shor. Teachers who use electronic portfolios 
incorporate principles of interaction and description into their evaluations; however, the 
social contexts surrounding their evaluations and the students’ perspectives on each 
other’s work have not yet altered the processes of assessment. 



* Baker argues that her 

thesis, that there is something wrong with our system of K-12 testing, does not flow from the same 
impulse as many such analyses. It is not developed as a critique of the factory model of education, 
the one that sees children as outputs and that is a vestige of the industrial age. It does not attack 
tests and their results as reductionist oddities. It does not compete with the findings of tests 
developed by the capital letters TESTING INDUSTRY against a sometimes more romantic view 
of the wisdom and accuracy of classroom teachers’ judgments of their students’ performance. 

Last, it will not deny that policymakers have the right and responsibility to demand testing 
programs that shed light on school progress and real policy options, and that such programs be 
developed on a schedule shorter than the Pleistocene era. (15) 
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Assessment remains an activity based on standards established before the students 
begin to communicate and establish their own interpretative community. In a fully- 
realized model of communication-based assessment, the four characteristics or attributes 
need to co-exist and interrelate. In existing electronic portfolio systems, they don’t. The 
rapid growth of electronic portfolios as tools for assessing not only student writing but 
also student learning across the disciplines in the last five years marks the beginnings of a 
shift in assessment theory and practice. 

Electronic Portfolios 

In 2002, we are still in the early stages of seeing how this shift will play out — and 
indeed, one of the questions that needs to be asked is whether assessment will shift at all 
or simply be more of the same. Much of the work with electronic portfolios has been 
based on print-based portfolio assessment work (often in writing programs) and theories 
of performance assessment. Electronic portfolios are used in places such as Alverno 
College and Kalamazoo College to present and assess student learning over the course of 
a college career. Electronic portfolios are used in schools of education at the University 
of Alaska, University of Virginia, and Ball State University as means to prepare teachers* 
in-training for the presentation of their own professional electronic portfolios. The ideal 
at Alaska, Virginia, and Ball State seems to be that the creation of teaching portfolios 
may drive the use of portfolios in K-12 education, and provide a basis forjudging teacher 
accountability in relationship to student performance on standardized assessments. 

Samples: Hart-Davidson, RPI; Rice, Texas Tech; Syverson, Texas; and Yancey, 
Clemson 
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I want to pause for a moment, and make four quick references to some outstanding work 
on electronic portfolios. These four sources, and the electronic portfolios that they link 
to, provide samples that have been crucial for me in testing out how the concepts of 
distributive, interactive, descriptive, and situated assessment are developing: 



• Bill Hart-Davidson’s work at Rensselaer Polytechnic Institute. An overview of 
his work is presented in “Electronic Portfolios & Outcomes- Based Assessment: A 
Case Study in Building an XML Object-Model for Decision Support”: 
http://www.rpi.edu/~hartdw/cec.ppt 

• Rich Rice’s dissertation “Teaching and Learning Frist-Year Composition with 
Digital Portfolios”: http://english.ttu.edu/rice/dissertation.pdf 

• M.A. Syverson’s work on the Learning Record online at the University of Texas: 
http://www.cwrl.utexas.edu/~syverson/olr/ 

• Kathleen Blake Yancey’s Introduction to the student portfolio section of the 
AAHE volume, Electronic Portfolios: 

http://aahe.ital.utexas.edU/electronicportfolios/TOC.html#yancey_introduction 



Electronic Portfolios and the Emerging Paradigms of Assessment 

Narrative or descriptive assessment has a history within American higher 
education. The University of California at Santa Cruz and Evergreen College have 
continued to employ narrative assessments instead of reductive letter grades for 
evaluation across the curriculum. In the context of distributive and interactive 
assessment in computer-mediated writing instruction, descriptive evaluation becomes 
even more important. As peer evaluators, students must be presented with a method of 
commenting on each other’s work that does not simply reinstate the standards- or 
criterion-based method of teacher commentary they have been previously exposed to. 
Peter Elbow defines “criterion-based feedback [as] the kind of feedback most people are 
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accustomed to — what they’ve usually gotten from teachers — and so it’s the kind of 
feedback that comes most naturally to people’s lips when you ask them for feedback” 
(Writing With Power 243). Since we do not yet fully understand what constitutes an 
effective rhetorical move in a computer-mediated composition, we can learn a lot by 
asking our students to describe both the criteria they are applying to electronic 
compositions and the ways in which they react to different electronic compositions. 

Theories of composing that view writing as part of an activity system or 
knowledge ecology propose models where the writing process is never singular but rather 
situated within the interactions of multiple agents and multiple environments. Students 
interact with one another; students interact with their teachers; students apply ideas about 
effective writing learned from previous teachers, previous writing experiences; students 
work within and against genre expectations; students compose within physical 
environments — in public computer labs, on the library, in their dorm rooms. All of these 
factors — and more — shape the work of writing. The complexity of causal relationships 
and interactions that knowledge ecologies and activity theories attribute to composing 
processes do not lead to a paralysis in terms of research, however. 

Nationally the uses of electronic portfolios both inside and outside of writing 
programs illustrate aspects of the evaluative methods I am advocating: distribution, 
interaction, description, and situated. By teasing out the theoretical basis for these 
methods of evaluation as well as their partial and different realizations in a variety of 
locations, we come to realize that we have reached a point in terms of available 
information technologies where we could — and need to — implement assessment methods 
that reflect the hybrid complexities of student language and student learning. The need to 



implement these methods of evaluation, as Eva Baker points out, is not simply a liberal or 
radical reaction to the dehumanizing effects of testing and assessment. Rather the 
pressures brought to bear by changes in the media used for literacy upon our means of 
reading, writing, learning, and evaluating are driving this re-evaluation of methods of 
assessment. A refusal to grapple with these issues will produce numerous failures in 
terms of assessment systems as well as in terms of students’ lives. This re-evaluation is 
not an abstract and self-contained process. It involves all sorts of tendencies and 
agencies working out of different motivations and positions. The question of how to 
assess student work as literacy practices and media are changing is not just a question 
asked by teacher-researchers — it is also a question raised by advocates of raised standards 
and mandated assessments, business coalitions lobbying for higher standards, and boards 
of education and trustees. Grounding the development of evaluative methods for 
computer-mediated writing instruction and electronic portfolio assessment in processes 
that distribute responsibility among a variety of readers and situate assessment within 
local, social contexts could eliminate the problems of mass, large-scale assessments and 
the decontextualized pedagogies they encourage. 
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